In a high-performance computer environment, for example an enterprise software application environment or a cloud-based environment, messaging typically plays an important role.
Generally described, messaging enables loosely-coupled software components (for example, enterprise software applications or cloud-based components) to communicate with each other. In a messaging system, messages are generated/produced by producers, and are delivered to a destination at a message broker, from where they can be received/consumed by consumers. Typically, the message broker operates as a server, and supports advanced features such as load-balancing and failover. A client application may include one or more producers and/or consumers, which can connect to the message broker to respectively produce and/or consume message data.
Scalability and elasticity are desirable characteristics of any messaging system, particularly those intended to support multiple tenants within a cloud-based environment. To address this, techniques such as clustering and data partitioning can be used. For example, the system can use a cluster of message brokers, together with a number of partitions to store message data produced by producers. Typically these partitions are considered permanent or “fixed partitions”. In the event of a broker failure, partitions owned by the failed broker are taken over by another broker. When the failed broker comes back, its previously owned partitions must be moved back in order to serve clients. Moving partitions can incur high cost in a messaging system not only due to data migration but also due to restoring message routing states in both broker and failover clients. This cost is necessary when load balancing applications that have a message ordering requirement for their produced messages, even during failover, because producers for these type of applications need to follow the partitions, or stick to the partitions, that they are producing messages to, in order to preserve message ordering in the event of failover. However, the need to migrate fixed partitions may incur a high computational cost with respect to data persistence and message routing, which negatively affects scalability and elasticity.
In some environments, applications have no requirement on message ordering. Load balancing of these applications among message brokers in a cluster does not have to incur partition migration. New “on-demand partitions” can be created on demand to serve clients, for example when a failed broker comes back or a new broker is added to the cluster. These on-demand partitions can be purged or removed when all messages in them are consumed by consumers.